Method and System for Anatomical Labels Generation

ABSTRACT

The present disclosure relates to a method and a system for generating anatomical labels of an anatomical structure. The method includes receiving an anatomical structure with an extracted centerline, or a medical image containing the anatomical structure with the extracted centerline; and predicting the anatomical labels of the anatomical structure based on the centerline of the anatomical structure, by utilizing a trained deep learning network. The deep learning network includes a branched network, a Graph Neural Network, a Recurrent Neural Network and a Probability Graph Model, which are connected sequentially in series. The branched network includes at least two branch networks in parallel. The method in the disclosure can automatically generate the anatomical labels of the whole anatomical structure in medical image end to end and provide high prediction accuracy and reliability.

CROSS REFERENCE TO RELATED APPLICATION

This application is based on and claims the priority of U.S. ProvisionalApplication No. 63/178,894, filed Apr. 23, 2021, which is incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to the technical field of processing andanalyzing medical data and medical image, and more specifically, to amethod and system for generating anatomical labels of an anatomicalstructure captured in a medical image.

BACKGROUND

Automatically identifying and assigning correct anatomical labels toanatomical structures facilitates the precise diagnosis. However, themorphology and topology of these anatomical structures vary widelyindividually. Therefore, the challenge of automatic anatomical structurelabeling arises from the large individual variability of coronaryanatomy, for example, especially some of the secondary branches emergingfrom the main branches.

Some of prior art estimate labels by using learning-based methods.However, these previous methods are not reliable to deal with the sceneof large individual variability as either the model is not end-to-end orthe model requires pre-defined features. Besides, the previous methodsdo not model for the correlation of each label. The above two pointsgreatly limit the performance such as accuracy and robustness of dealingwith the scene of large individual variability.

SUMMARY

Certain embodiments may provide a method and a system for generatinganatomical labels of the anatomical structure. Such a method and systemmay automatically generate anatomical labels of the whole anatomicalstructure in medical images in an end-to-end manner, by utilizing atrained deep learning network. The method and system may also providestrong robustness, higher prediction accuracy and reliability despite ofa large individual variability in coronary anatomy. The disclosedembodiments are provided to solve at least the technical problemsmentioned above.

According to a first aspect of the present disclosure, there is provideda method for generating anatomical labels of an anatomical structure.The method begins with receiving an anatomical structure with anextracted centerline, or a medical image containing the anatomicalstructure with the extracted centerline. Next, the method includespredicting the anatomical labels of the anatomical structure based onthe centerline of the anatomical structure, by utilizing a trained deeplearning network/. The deep learning network includes a branchednetwork, a Graph Neural Network, a Recurrent Neural Network and aProbability Graph Model which are connected sequentially in series. Thebranched network includes at least two branch networks in parallel.

According to a second aspect of the present disclosure, there isprovided a system for generating anatomical labels of the anatomicalstructure. The system includes an interface configured to receive ananatomical structure with an extracted centerline, or a medical imagecontaining the anatomical structure with the extracted centerline. Thesystem further includes at least one processor configured to predict theanatomical labels of the anatomical structure based on the centerline ofthe anatomical structure, by utilizing a trained deep learning network.The deep learning network includes a branched network, a Graph NeuralNetwork, a Recurrent Neural Network and a Probability Graph Model whichare connected sequentially in series. The branched network includes atleast two branch networks in parallel.

According to the third aspect of the present disclosure, there isprovided a non-transitory computer-readable storage medium, withcomputer-executable instructions stored thereon. The instructions, whenexecuted by a processor, cause the processor to perform a method forgenerating anatomical labels of the anatomical structure by utilizing acomputer, corresponding to the method described above.

Embodiment disclosed herein provide an end-to-end precise prediction ofthe anatomical structure, based on the centerline graph constructed bysampling points of the centerline of the anatomical structure, byconsidering various features such as the geometric feature and the imagefeature of the centerline graph, utilizing multiple branch networksprovided in parallel and implementing a joint embedding of multiplefeatures by a Graph Neural Network. They enable a more reasonable androbust division of the whole anatomical structure from the perspectiveof global optimization by modeling for the relationship betweenanatomical labels, utilizing a Probability Graph Model. Therefore, themethods in the present disclosure can realize automatic end-to-endprediction of the anatomical labels of the anatomical structure withhigher accuracy and robustness, and are also more suitable for theprediction of the anatomical labels of the anatomical structures withgreater individual variability. Accordingly, the diagnosis accuracy anddiagnosis efficiency of doctors may be improved.

The above general description and the following detailed description areexemplary and explanatory only, and are not intended to limit theclaimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, likereference numerals may describe similar components in various views.Like reference numerals having letter suffixes or different lettersuffixes may represent different instances of similar components. Thedrawings illustrate generally, by way of example, but not by way oflimitation, various embodiments, and together with the description andclaims, serve to explain the disclosed embodiments. Such embodiments aredemonstrative and not intended to be exhaustive or exclusive embodimentsof the present method, device, system and non-transitory computerreadable medium with instructions for implementing the method.

FIG. 1 illustrates a schematic diagram of exemplary anatomical labels ofthe coronary artery, according to an embodiment of the presentdisclosure.

FIG. 2 illustrates a flowchart of a method for generating anatomicallabels of the anatomical structure, according to an embodiment of thepresent disclosure.

FIG. 3 illustrates a schematic diagram of a procedure of predictinganatomical labels by utilizing a trained deep learning network,according to an embodiment of the present disclosure.

FIG. 4 illustrates a schematic diagram of another procedure ofpredicting anatomical labels by utilizing a trained deep learningnetwork, according to an embodiment of the present disclosure.

FIG. 5 illustrates a schematic block diagram of a system for generatinganatomical labels of the anatomical structure, according to anembodiment of the present disclosure.

FIG. 6 illustrates a schematic workflow diagram of a system forgenerating anatomical labels of the anatomical structure, according toan embodiment of the present disclosure.

DETAILED DESCRIPTION

In order to enable those skilled in the art to understand the presentinvention better, the embodiments of the present invention will bedescribed in detailed below with reference to the drawings, but not as alimitation of the present invention.

FIG. 1 illustrates a schematic diagram of exemplary anatomical labels ofthe coronary artery according to the embodiment of the presentdisclosure. It is to be noted that the technical scheme of the presentdisclosure will be explained below by taking the anatomical structure asthe coronary artery or the anatomical structure contained in theacquired medical image as coronary artery as an example, but the presentdisclosure is not limited to this.

Coronary artery is the artery that supplies blood to the heart. Itoriginates from the aortic sinus at the root of the aorta and is dividedinto left and right branches and runs on the surface of the heart.According to Schlesinger's classification principle, the distribution ofthe coronary artery may be divided into right dominant type, balancedtype and left dominant type. Different dominant types of the coronaryartery may contain different anatomical structures, and there are manyrules for labeling anatomical structures. For example, according to theclassification of American heart association (AHA), the coronaryarteries may be roughly divided into 15 segments, that is, 15 types ofthe anatomical labels. FIG. 1 is a specific manner of anatomicallabeling of coronary arteries on the basis of AHA classification. Therules for segmenting and labeling the anatomical structure of coronaryarteries are shown in Table 1.

TABLE 1 Anatomical labels of coronary artery and its labeling rules inFIG. 1 Anatomical labels Labeling rules  1. pRCA: Proximal right Ostiumof the RCA to one-half the coronary artery (RCA) distance to the acutemargin of heart coronary artery (RCA)  2. mRCA: Middle RCA End ofproximal RCA to the acute margin of heart  3. dRCA: Distal RCA End ofmid RCA to origin of the PDA (posterior descending artery)  4. R-PDA:PDA-R PDA from RCA LAD branch: 5-6-7-8  5. LM: Left main (LM) Ostium ofLM to bifurcation ofLAD(left anterior descending artery)and LCx(leftcircumflex artery)  6. pLAD: Proximal LAD End of LM to the first largeseptal or D1(first diagonal; >1.5 mm in size) whichever is most proximal 7. mLAD: Mid LAD End of proximal LAD to one-half the distance to theapex  8. dLAD: Distal LAD End of mid LAD to end of LAD D branch: 9, 10 9. D1: First diagonal branch First diagonal branch D1 10. D2: Seconddiagonal Second diagonal branch D2 branch 11. pCx: Proximal LCx End ofLM to the origin of the OM1 (first obtuse marginal) 12. OM1: OM1 FirstOM1 traversing the lateral wall of the left ventricle 13. LCx: Middleand distal Traveling in the atriove ntricular groove, LCx distal to theOM1 branch to the end of the vessel or origin of the L-PDA(left PDA) 14.OM2: Second marginal Second marginal OM2 OM2 15. PDA-L: L-PDA PDA fromLCx 16. PLB-R: R-PLB PLB from RCA 17. RI: Ramus intermedius Vesseloriginating from the left main between the LAD and LCx in case of atrifurcation 18. PLB-L: L-PLB PLB from LCx

The anatomical structure of the coronary artery shown in FIG. 1 may varygreatly in different individuals. For example, the 14th segment in thedrawing, that is the second marginal OM2, may be absent in someindividuals, and so on, In addition, the labeling rules of theanatomical labels can also include the relationship between anatomicallabels. Taking Table 1 as an example, LAD branch is connected by segment5 (LM: Left Main)—segment 6 (pLAD: Proximal LAD)—segment 7 (mLAD: MidLAD)—segment 8 (dLAD: Distal LAD). D branch includes segment 9 (D1:First diagonal branch) and segment 10 (D2; Second diagonal branch),while segment 9 originated from segment 6 or 7, and segment 10originated from segment 8 or the transition part of segment 7 and 8. Inother embodiments, the D branch may also include segment 9 a (D1a: Firstdiagonal a, not shown) and segment 10 a (D2a: Second diagonal a, notshown), etc. In this case, segment 9 a may be additional first diagonaloriginating from segment 6 or 7, before segment 8 (the distal segment ofthe anterior descending branch). In some embodiments, the methodaccording to the present disclosure can comprehensively consider theabove bidirectional constraint relationship between anatomical labelswhen determining the anatomical labels of various parts of theanatomical structure, so that the prediction of the anatomical labels ofthe whole anatomical structure may be more accurate and reliable fromthe perspective of global optimization.

It is to be noted that the anatomical structure according to theembodiment of the present disclosure is not necessarily a coronaryartery, but also any vessel, respiratory tract, mammary duct, etc.,especially an anatomical structure with a multi-branched tree, which isnot listed here.

FIG. 2 illustrates a flowchart of a method for generating the anatomicallabels of the anatomical structure, according to the embodiment of thepresent disclosure.

First, at step S101, an anatomical structure with an extractedcenterline or a medical image containing the anatomical structure withthe extracted centerline may be received. In some embodiments, it isalso possible to receive an anatomical structure without the extractedcenterline or a medical image containing the anatomical structurewithout the extracted centerline, and extract the centerline of theanatomical structure by using any algorithm for extract the centerline,at the same time, some useful data information, such as the segmentationmask, may be extracted together. The above process for centerlineextracting may be implemented automatically, semi-automatically, ormanually, and any suitable algorithm or method may be adopted. Thepresent disclosure is not limited to this.

Next, at step S102, the anatomical labels of the anatomical structuremay be predicted based on the centerline of the anatomical structurereceived or extracted at step 101, by utilizing a trained deep learningnetwork. In some embodiments, the above deep learning network mayinclude a branched network including at least two branch networks, aGraph Neural Network, a Recurrent Neural Network and a Probability GraphModel. These networks are connected sequentially in series, with the atleast two branch networks within the branched network in parallel.

In some embodiments, when predicting the anatomical labels based on thecenterline of the anatomical structure, a graph representation of thecenterline of the anatomical structure may be first constructed based onthe centerline of the anatomical structure. For example, it is possibleto sample for the centerline and take each sampling point as a node(hereinafter represented by V) of a centerline graph (hereinafterrepresented by G) of the centerline, and take a line segment on thecenterline connecting each pair of adjacent nodes as an edge(hereinafter represented by E) of the centerline graph.

In some embodiments, for each node v_(i) in V, the associated physiccoordinates, image patches and any other useful information may beextracted as the representation of the node v_(i).

In some embodiments, for example, the edge e_(i) in E may be representedas one undirected edge or two directed edges, where the directed edgemay bring more information than undirected edges, especially may be usedto represent the bidirectional constraint relationship between two nodesconnected by e_(i). For example, in the coronary artery tree structure,information can be propagated from the root of the tree to the terminal,and it can also be propagated from the opposite direction (from terminalto the root).

Based on the settings of the set V of nodes and the set E of edges, thecenterline graph G can be expressed as: G=(V, E), where the node v_(i) ∈V in V corresponds to the feature vectors or embedding of the points onthe centerline, and e_(i) ∈ E corresponds to the directed or undirectededges between the points. i∈[1, . . . , N], j∈[1, . . . ,N−1], where Nis the node number of the centerline graph G.

Once modeling the centerline graph of the anatomical structure has beenfinished, it is possible to predict the anatomical labels of theanatomical structure based on the centerline graph G, by utilizing thetrained deep learning network. The specific procedure will be describedin detail with reference to FIG. 3.

The method for determining the anatomical labels in the disclosedembodiment takes the anatomical structure and the medical imagecontaining the anatomical structure as input, and realizes end-to-endprediction of the anatomical labels by using multiple of deep learningnetworks such as deep neural networks combined in parallel and inseries, which can learn the anatomical feature for identifying theessence of the anatomical structure such as arteries without anyartificially defined standards and features. Compared with the priorart, the pre-defined discrete feature extraction module s no longerneeded, instead, from the perspective of global joint optimization, theabove deep learning network is learned and used as a whole, and theanatomical labels of each part in the anatomical structure may bedirectly output at the output end of the deep learning network. With theincrease of the amount of training data, the performance of the deeplearning network in terms of accuracy and robustness will also beimproved.

In some embodiments, first, the first branch network 301 can extract thecoordinate information of each node v_(i) based on the centerline graphG in step S3011, and use the extracted coordinate information as theinput for embedding the geometric feature. Since the coordinateinformation may be used as a point cloud, in step S3012, for example,any point cloud network such as PointNet, PointNet++ may be used toencode the coordinate information and generate the geometric featureembedding of each node. The present disclosure is not particularlylimited to the point cloud network used by the first branch network 301,as long as the geometric feature embedding of each node may be generatedbased on the coordinate information of each node in the centerline graphG.

In some embodiments, there may be one or more other branch networksprovided in parallel with the above first branch network 301, such asthe second branch network 302 shown in FIG. 3. In some embodiments, thesecond branch network 302 may synchronize with the first branch network301, and in step S3021, extract the 2D/3D image patch or the mask patchcorresponding to each node v_(i) based on the center line graph G, anduse the extracted image patch/mask patch as input for embedding theimage feature. In some embodiments, for example, the optimized windowwidth may be selected according to different anatomical parts, so as tocomplete the extraction of the image patch/mask patch, etc. The presentdisclosure is not specifically limit to this. In some embodiments, forexample, in step S3022, the second branch network 302 may adopt any deeplearning network based on CNN(Convolutional Neural Network), GCN (GraphConvolutional Neural Network), RNN(Recurrent Neural Network) orMLP(Multilayer perceptron), such as ResNet, VGG, etc., encode theimage/mask information of each node and generate the image/mask featureembedding of each node.

As an example only, in case of the second branch network 302 selects GCNas the deep learning network for embedding the image feature, GCN cangeneralize the architecture of CNN to non-Euclidean domains such as thegraph. The graph convolution defines convolutions directly on the graph,operating on spatially close neighbors. Formally, Z=GCN(X, A), where X∈RN×C, is the input, N is the node number, and C is the dimension of thefeature embedding, A is an adjacent matrix to denote if there are edgesbetween nodes, in the embodiment of the present disclosure, A may bedetermined by the centerline graph, Z is the output of the GCN. Itshould be note that other common methods used in CNN can also be used inGCN, such as skipping connection or attention. In addition, in someembodiments, the second branch network 302 can also select otheradvanced GNN variants, for example, gated GNN method. In otherembodiments, it can also use the gate mechanism like GRU or LSTM in thepropagation step to improve the long-terra propagation of informationacross the graph structure. For example, if the edges of the graph aredirectional, by using a gate mechanism, the parent node can selectivelyincorporate information from each child node. More specifically, eachgraph unit (could be a GRU or LSTM unit) includes input and outputgates, a memory cell, and a hidden state. Instead of a single forgetgate, each graph unit includes one forget gate for each child node. Theabove graph unit could be any RNN unit such as LSTM, GRU, CLSTM, CGRU,etc.

As mentioned above, the present disclosure is not particularly limitedto the deep learning network and feature embedding method used by thesecond branch network 302, as long as the image information coding canbe realized, and the image/mask feature embedding of each node can begenerated based on the image patch/mask patch corresponding to each nodein the centerline graph G.

After obtaining the geometric feature embedding and the image featureembedding of each node of the centerline graph G by the first branchnetwork 301 and the second branch network 302 respectively, next, theGraph Neural Network 303 may be used to integrate the geometric featureembedding and the image feature embedding of each node in step S3031 toobtain the joint feature embedding of each node. The method ofintegrating two or more feature embeddings is not specifically limitedhere, and it may simply concatenate the feature embedding of eachbranch, or combining the feature embeddings of multiple branchesaccording to the predetermined weight to generate the joint featureembedding, which are not listed here.

Next, the joint feature embedding of each node output by the GraphNeural Network 303 can be the input of the Recurrent Neural Network 304,and the Recurrent Neural Network 304 can be used to generate theanatomical label corresponding to each node in the centerline graph G ofthe anatomical structure by utilizing the Recurrent Neural Network 304.The above Recurrent Neural Network 304 may adopt any one of LSTM, GRU,CLSTM, CGRU, or a variation based on it. The present disclosure is notlimited to this.

Further, the cell division of the whole anatomical structure and theanatomical label corresponding to each cell may be generated, forexample, by induction or clustering, based on the anatomical labelcorresponding to each node in the centerline graph G and according tothe anatomical label corresponding to each node by utilizing theProbability Graph Model 305. The above Probability Graph Model may alsobe other models besides CRF, such as MRF (Markov Random Field) orhigh-order CRF, or other variations based on this. The presentdisclosure is not limited to this.

According to the method for determining the anatomical labels in theembodiment of the present disclosure as shown in FIG. 3, the geometricfeature or the image feature in the centerline graph constructed basedon the anatomical structure not only may be considered independently,but also may be considered jointly being integrated into a joint featureembedding vector by using GNN network. The joint feature embeddingvector may improve the prediction accuracy of the deep learning network,regardless the joint feature embedding vector is generated by simpleconcatenating or by weighted fusion.

In some embodiments, the centerline may be divided into a plurality ofcells at first, and after the joint feature embedding of each node isgenerated by utilizing the Graph Neural Network, performing cell levelaverage pooling for the joint feature embedding of each node based onthe divided cells of the centerline to generate cell level features.Then, generating the cell level anatomical labels of the centerlinegraph based on the cell level features, by utilizing the RecurrentNeural Network. Finally, generating the anatomical labels of theanatomical structure based on the cell level anatomical labels, byutilizing the Probability Graph Model.

FIG. 4 illustrates a schematic diagram of another procedure ofpredicting the anatomical labels by utilizing the trained deep learningnetwork according to an embodiment of the present disclosure. FIG. 4 issimilar to FIG. 3 in that the deep learning network 300 includescomponents of the first branch network 301, the second branch network302, the Graph Neural Network 303, the Recurrent Neural Network 304 andthe Probability Graph Model 305, and the operations and steps in eachcomponent are basically the same. FIG. 4 is different from FIG. 3 inthat, after the Graph Neural Network 303 integrates the geometricfeatures and image features of each node and outputs the joint featureembedding of each node in step S3031, it is not directly sent to theRecurrent Neural Network 304, but a pooling unit 303′ is used, and instep S3032, cell level average pooling is performed for the jointfeature embedding of each node.

In some embodiments, in order to obtain more accurate anatomical labelsprediction results when sampling the centerline of the anatomicalstructure, the sampling is usually denser, the number of nodes islarger, and the correlation between adjacent nodes is larger. Inanatomical structure such as vessel, the vessel between two bifurcationsusually belong to the same vessel branch, that is, have the sameanatomical labels. Therefore, it can be set that in case of theanatomical structure is a vessel, each cell of the centerline is thevessel branch between two bifurcations. The centerline may be dividedinto cells manually, automatically, or semi-automatically, and thepresent disclosure is not limit to this.

Taking the coronary artery shown in FIG. I as an example, the vesselbranch between bifurcations may be taken as basic cell along thecenterline of the coronary artery, and performing cell level averagepooling for the centerline graph with the joint feature embedding. Thecell level features output by the pooling unit 303′ are input to theRecurrent Neural Network 304 after cell level average pooling. At thistime, the Recurrent Neural Network 304 will output the anatomical labelsequence of each cell based on the cell level features, which isdifferent from FIG. 3.

As shown in the previous FIG. 1 and Table 1, D branches such as D1 andD2 are located on the LAD branch consisting of 5-6-7-8. That is to say,for structured labeling tasks, it is beneficial to consider thecorrelations between labels in neighborhoods and the whole structure andjointly decode the best structure of labels for a given input structure.Therefore, in some embodiments, the Probability Graph Model (PGM) suchas the Conditional Random Field (CRF) may be used to jointly model thecorrelation of the labels, instead of decoding the anatomical labels ofeach cell independently, so that more reasonable and accurate anatomicallabels may be obtained from the perspective of the whole anatomicalstructure.

In some cases, the anatomical structure may change greatly fromindividual to individual. Using the method of the embodiment of thepresent disclosure to model the relationship between the anatomicallabels may deal with this change of individuals in a large degree. Underthe constraints of the relationship between anatomical labels, themethod according to the present disclosure can make accurate and robustpredictions of the anatomical labels in the whole anatomical structure,such as the whole blood vessel tree, by making use of these constraints,thereby helping doctors to accurately and efficiently respond toindividual differences and make accurate and reliable diagnosis fordifferent patients.

The embodiment according to the present disclosure also provides adevice for determining the anatomical labels of the anatomical structureby utilizing the computer, which includes a storage, at least oneprocessor, and computer-executable instructions stored on the storageand ran on the at least one processor, wherein the at least oneprocessor executes the steps of the method for determining theanatomical labels of the anatomical structure by utilizing the computerdescribed in the previous embodiments.

The embodiment of the present disclosure also provides a system fordetermining the anatomical labels of the anatomical structure byutilizing the computer. The system includes an interface, a modeltraining device and an image processing device, wherein the interface isused receive an anatomical structure with an extracted centerline, or amedical image containing the anatomical structure with the extractedcenterline required in the training phase, and/or receive the anatomicalstructure with the extracted centerline, or the medical image containingthe anatomical structure with the extracted centerline of which theanatomical labels are to be predicted in the prediction phase.

The system for determining the anatomical labels of the anatomicalstructure by utilizing the computer also includes the model trainingdevice, which is used to train the deep learning network in the methodfor determining the anatomical labels of the anatomical structure byutilizing the computer described in the previous embodiments in thetraining phase.

The system for determining the anatomical labels of the anatomicalstructure by utilizing the computer also includes the image processingdevice, which is used to perform the steps of the method for determiningthe anatomical labels of the anatomical structure by utilizing thecomputer described in the previous embodiments in the prediction phase.

FIG. 5 illustrates a schematic block diagram of the system fordetermining the anatomical labels of the anatomical structure byutilizing the computer according to the system of the embodiment of thepresent disclosure. The system 500 may include the model training device502 configured to train the deep learning network according to theembodiment of the present disclosure in the training phase, and theimage processing device 503 configured to perform the anatomical labelprediction task in the prediction phase. In some embodiments, the modeltraining device 502 and the image processing device 503 may be withinthe same computer or processing device.

In some embodiments, the image processing device 503 may be aspecial-purpose computer or a general-purpose computer. For example, theimage processing device 503 may be a customized computer that performsimage acquisition or image processing tasks in a hospital or a serverarranged in the cloud. The image processing device 503 may include acommunication interface 501, a memory 504, a storage 506, a processor508, and a bus 510. The interface 501, the memory 504, the storage 506,and the processor 508 are connected to the bus 510 and communicate witheach other through the bus 510.

The communication interface 501 may include a network cable connector, acable connector, a serial connector, a USB connector, a parallelconnector, a high-speed data transmission adapter such as optical fiber,USB 3.0, Thunder, etc., a wireless network adapter such as a WiFiadapter, a telecommunications (3G, 4G/LTE, etc.) adapter, etc. In someembodiments, the interface 501 receives the medical image containing theanatomical structure from the image acquisition device 505. In someembodiments, the interface 501 also receives the trained deep learningnetwork model from the model training device 502.

The image acquisition device 505 can acquire images in any imaging formin functional MRI (such as fMRI, DCE-MRI and diffusion MRI), cone-beamcomputed tomography (CBCT), spiral CT, positron emission tomography(PET), single photon emission computed tomography (SPECT), X-rayimaging, optical tomography, fluorescence imaging, ultrasound imagingand radiation field imaging, etc. or their combination. The disclosedmethod may be performed by a system that uses the acquired image to makeanatomical label prediction.

The memory 504/storage 506 may be a non-transitory computer-readablemedium, such as read only memory (ROM), random access memory (RAM),phase change random access memory (PRAM), static random access memory(SRAM), dynamic random access memory (DRAM), electrically erasableprogrammable read only memory (EEPROM), other types of random accessmemory (RAMs), flash disk or other forms of flash memory, cache,register, static memory, compact disc read-only memory (CD-ROM), digitalversatile disc (DVD) or other optical storage, magnetic tape or othermagnetic storage devices, or any other non-transitory medium that may beused to store information or instructions that may be accessed by thecomputer device, etc.

In some embodiments, the memory 504 may store the trained deep learningmodel and data, such as the centerline graph generated when the computerprogram is executed. In some embodiments, the storage 506 may storecomputer-executable instructions, such as one or more image processingprograms.

The processor 508 may be a processing device including one or moregeneral-purpose processing devices, such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit (GM), etc. Morespecifically, the processor may be a complex instruction set computing(CISC) microprocessor, a reduced instruction set computing (RISC)microprocessor, a very long instruction word (VLIW) microprocessor, aprocessor running other instruction sets, or a processor running acombination of instruction sets. The processor can also be one or morespecial processing devices, such as application specific integratedcircuit (ASIC), field programmable gate array (FPGA), digital signalprocessor (DSP), system on chip (SoC), etc. The processor 508 may becommunicatively coupled to the storage 506 and configured to executecomputer-executable instructions stored thereon.

The model training device 502 may be realized by hardware speciallyprogrammed by software that executes training processing. For example,the model training device 502 may include the same processor andnon-transitory computer readable medium as the image processing device503. The processor may be trained by executing the instructions of thetraining process stored in the computer-readable medium. The modeltraining device 502 may also include input and output interfaces tocommunicate with a training database, a network, and/or a userinterface. The user interface may be used to select the training dataset, adjust one or more parameters of the training process, select ormodify the framework of the learning model, and/or manually orsemi-automatically provide the prediction results related to theanatomical structure in the trained image.

FIG. 6 is a schematic workflow diagram of the system for determining theanatomical labels of the anatomical structure by utilizing the computeraccording to an embodiment of the present disclosure.

As shown in FIG. 6, the workflow of the system for determining theanatomical labels of the anatomical structure by utilizing the computeraccording to the embodiment of the present disclosure may be dividedinto the training phase and the prediction phase, and the specificworkflow is as follows:

In some embodiments, the training phase 61 is an off-line process. Atthis phase, first, the training image may be obtained in step S611, forexample, the medical image containing the anatomical structure with orwithout the extracted centerline. In case of the centerline of theanatomical structure is not extracted, any applicable centerlineextraction algorithm may be used to extract the centerline of theanatomical structure in the training image in step S612. Next, in stepS613, the deep learning network to be trained may be modeled. The deeplearning network is composed of at least two branch networks, the GraphNeural Network, the Recurrent Neural Network and the Probability GraphModel which is connected sequential in series, wherein the at least twobranch networks are provided in parallel. In step S613, the imagepatch/mask patch and coordinates and other data informationcorresponding to each sampling point may be automatically extracted fromthe centerline by using the graph representation algorithm, create thecenterline graph of the anatomical structure, and embed the feature ofeach node in the centerline graph. In the training phase, the groundtruth of the anatomical labels may be obtained in step S614, or at thisphase, the system assembles a database of anatomical structure trainingdata labeled with the ground truth. Next, in step S615, the modeled deeplearning network may be trained based on the centerline graph afterfeature embedding and the ground truth of the anatomical labels, and thetrained deep learning network may be obtained in step S616. Whentraining the end-to-end deep learning network model, gradient-basedmethods (such as SGD, Adam, et ay he used to optimize the objectivefunction J relative to all parameters on the training data set. Theparameter (θ) of the deep learning network model may be optimized byminimizing the mean square deviation between the ground truth y and thepredicted value output ŷ of each node on the centerline graph.Especially, the parameter (θ) may be optimized for the training set D tominimize the objective function J, where J may be any classificationloss or AUC loss.

The prediction phase 62 may be an online process. In some embodiments,first, in step S621, a new test image of the anatomical labels to bepredicted may be received, which should include the anatomical structurewith or without the extracted centerline. In case of the centerline ofthe anatomical structure in the received test image is not extracted,the centerline of the anatomical structure in the test image may beextracted in step S622. Then, in step S623, the anatomical labels of thewhole anatomical structure in the new test image may be calculated byutilizing the deep learning network trained in the training phase 61.

Various modifications and changes can be made to the method, device andsystem of the present disclosure. Other embodiments can be derived bythose skilled in the art in view of the description and practice of thedisclosed system and related methods. Each claim of the presentdisclosure can be understood as an independent embodiment, and anycombination between them can also be used as an embodiment of thepresent disclosure, and these embodiments are deemed to be included inthe present disclosure.

The examples merely regarded as exemplary only, and the true scope isindicated by the appended claims and their equivalents.

What is claimed is:
 1. A computer-implemented method for generatinganatomical labels of an anatomical structure, comprising: receiving ananatomical structure with an extracted centerline, or a medical imagecontaining the anatomical structure with the extracted centerline; andpredicting, by at least one processor, the anatomical labels of theanatomical structure based on the centerline of the anatomicalstructure, by utilizing a trained deep learning network, wherein thedeep learning network comprises a branched network, a Graph NeuralNetwork, a Recurrent Neural Network and a Probability Graph Model, whichare connected sequentially in series, wherein the branched networkcomprises at least two branch networks in parallel.
 2. Thecomputer-implemented method according to claim 1, wherein predicting theanatomical labels of the anatomical structure based on the centerline ofthe anatomical structure, by utilizing a trained deep learning networkfurther comprises: sampling the centerline to form a centerline graph ofthe anatomical structure, wherein each sampling point is a node of thecenterline graph and each line segment on the centerline connecting eachpair of adjacent nodes is an edge of the centerline graph; andpredicting the anatomical labels of the anatomical structure based onthe centerline graph, by utilizing the trained deep learning network. 3.The computer-implemented method according to claim 2, wherein the atleast two branch networks include a first branch network and a secondbranch network, and predicting the anatomical labels of the anatomicalstructure based on the centerline graph, by utilizing the trained deeplearning network further comprises: generating a geometric featureembedding of each node based on coordinate information of each node, byutilizing the first branch network; generating an image featureembedding of each node based on an image patch or an mask patchcorresponding to each node, by utilizing the second branch network;generating a joint feature embedding of each node based on the geometricfeature embedding and the image feature embedding, by utilizing theGraph Neural Network; generating an anatomical label corresponding toeach node in the centerline graph based on the joint feature embedding,by utilizing the Recurrent Neural Network; and generating the anatomicallabels of the anatomical structure based on the anatomical labelcorresponding to each node in the centerline graph, by utilizing theProbability Graph Model.
 4. The computer-implemented method according toclaim 3, wherein predicting the anatomical labels of the anatomicalstructure based on the centerline graph, by utilizing the trained deeplearning network further comprises: dividing the centerline into aplurality of cells; after generating a joint feature embedding of eachnode by utilizing the Graph Neural Network, performing a cell levelaverage pooling for the joint feature embedding of each node based onthe divided cells of the centerline to generate cell level features;generating the cell level anatomical labels of the centerline graphbased on the cell level features, by utilizing the Recurrent NeuralNetwork; and generating the anatomical labels of the anatomicalstructure based on the cell level anatomical labels, by utilizing theProbability Graph Model.
 5. The computer-implemented method according toclaim 4, wherein the anatomical structure is a vessel, and each cell ofthe centerline corresponds to a vessel branch between two bifurcationsof the vessel.
 6. The computer-implemented method according to claim 3,wherein the first branch network is a point cloud neural network.
 7. Thecomputer-implemented method according to claim 3, wherein the secondbranch. network is a CNN, a RNN, or an MLP.
 8. The computer-implementedmethod according to claim 3, wherein the Recurrent Neural Network is anLSTM, a GRU, a GLSTM, or a CGRU.
 9. The computer-implemented methodaccording to claim 1, wherein the anatomical structure comprises atleast one of the vessel, a respiratory tract, or a mammary duct.
 10. Thecomputer-implemented method according to claim 2, wherein the edges ofthe centerline graph are directed edges.
 11. The computer-implementedmethod according to claim 1, further comprising: receiving training datacomprising a sample anatomical structure with an extracted samplecenterline or a sample medical image containing the sample anatomicalstructure with the extracted sample centerline and a ground truth of theanatomical labels of the sample anatomical structure; and training thedeep learning network by performing a joint optimization of modelparameters of the at least two branch networks, the Graph NeuralNetwork, the Recurrent Neural Network and the Probability Graph Modelusing the training data.
 12. A system for generating anatomical labelsof an anatomical structure, comprising: an interface configured toreceive an anatomical structure with an extracted centerline, or amedical image containing the anatomical structure with the extractedcenterline; and at least one processor configured to predict theanatomical labels of the anatomical structure based on the centerline ofthe anatomical structure, by utilizing a trained deep learning network,the deep learning network comprises a branched network, a Graph NeuralNetwork, a Recurrent Neural Network and a Probability Graph Model whichare connected sequentially in series, wherein the branched networkcomprises at least two branch networks in parallel.
 13. The systemaccording to claim 12, wherein to predict the anatomical labels of theanatomical structure based on the centerline of the anatomicalstructure, by utilizing a trained deep learning network, the at leastone processor is further configured to: sample the centerline to form acenterline graph of the anatomical structure, wherein each sample pointis a node of the centerline graph and each line segment on thecenterline connecting each pair of adjacent nodes is an edge of thecenterline graph; and predict the anatomical labels of the anatomicalstructure based on the centerline graph, by utilizing the trained deeplearning network.
 14. The system according to claim 13, wherein the atleast two branch networks include a first branch network and a secondbranch network, and to predict the anatomical labels of the anatomicalstructure based on the centerline graph, by utilizing the trained deeplearning network, the at least one processor is further configured to:generate a geometric feature embedding of each node based on coordinateinformation of each node, by utilizing the first branch network;generate an image feature embedding of each node based on an image patchor an mask patch corresponding to each node, by utilizing the secondbranch network; generate a joint feature embedding of each node based onthe geometric feature embedding and the image feature embedding, byutilizing the Graph Neural Network; generate an anatomical labelcorresponding to each node in the centerline graph based on the jointfeature embedding, by utilizing the Recurrent Neural Network; andgenerate the anatomical labels of the anatomical structure based on theanatomical label corresponding to each node in the centerline graph, byutilizing the Probability Graph Model.
 15. The system according to claim14, wherein to predict the anatomical labels of the anatomical structurebased on the centerline graph, by utilizing the trained deep learningnetwork, the at least one processor is further configured to: divide thecenterline into a plurality of cells; generate a joint feature embeddingof each node by utilizing the Graph Neural Network, performing a celllevel average pooling for the joint feature embedding of each node basedon the divided cells of the centerline to generate cell level features;generate the cell level anatomical labels of the centerline graph basedon the cell level features, by utilizing the Recurrent Neural Network;and generate the anatomical labels of the anatomical structure based onthe cell level anatomical labels, by utilizing the Probability GraphModel.
 16. The system according to claim 15, wherein the anatomicalstructure is a vessel, and each cell of the centerline is a vesselbranch between two bifurcations of the vessel.
 17. The system accordingto claim 14, wherein the first branch network is a point cloud neuralnetwork.
 18. The system according to claim 14, wherein the second branchnetwork is a CNN, a RNN, or an MLP.
 19. The system according to claim14, wherein the Recurrent Neural Network is an LSTM, a GRU, a GLSTM, ora CGRU.
 20. A non-transitory computer-readable storage medium storingcomputer-executable instructions thereon, wherein thecomputer-executable instructions, when executed by a processor, causethe processor to perform a method for generating anatomical labels of ananatomical structure, the method comprising: receiving an anatomicalstructure with an extracted centerline, or a medical image containingthe anatomical structure with the extracted centerline; and predicting,by at least one processor, the anatomical labels of the anatomicalstructure based on the centerline of the anatomical structure, byutilizing a trained deep learning network, wherein the deep learningnetwork comprises a branched network, a Graph Neural Network, aRecurrent Neural Network and a Probability Graph Model, which areconnected sequentially in series, wherein the branched network comprisesat least two branch networks in parallel.